QTM 385 - Experimental Methods

Lecture 2 - The Research Design Process

Danilo Freire
danilo.freire@emory.edu

Emory University

Hello, everyone! 👋
Nice to meet you again! 😊

Recap and lecture overview 📚

Welcome back! 🎉

About the course

Last week

  • We saw a brief history of experimental methods and their main characteristics
    • Key milestones from Petrarch (1394) to modern RCTs
  • We also saw the main types of experiments
    • Lab Experiments: Conducted in controlled environments with high internal validity
    • Field Experiments: Take place in real-world settings, offering higher external validity
    • Natural Experiments: Utilise naturally occurring variations, providing less control but valuable insights
  • Finally, we briefly discussed the distinction between correlation and causation
  • Key tools: R/Python, GitHub, Quarto, and Jupyter Notebooks
  • Assignments: Problem sets (50%), pre-analysis plan (20%), and final project (30%), with a late policy and collaboration guidelines

Course website and materials

Click on the image to access the course website! 🌐

https://danilofreire.github.io/qtm385/

Textbooks

Required: Alan Gerber and Donald Green - Field Experiments

Recommended: Douglas C. Montgomery - Design and Analysis of Experiments

Lots of additional readings included in the syllabus 😊

Today’s lecture 📅

The research design process: Questions and credibility

  • What makes a research question good?
  • The importance of theory (there’s no such thing as a theory-free observation!)
  • Designing and selecting your treatment
  • How to interpret your findings carefully
  • The importance of transparency and reproducibility
  • The research design process
  • EGAP research design form: A guide to help you think through the key components of your research design
  • Introduction to DeclareDesign and the MIDA framework
  • Brief overview of pre-registration and pre-analysis plans

But first…

Let’s do an experiment right now! 😄

Instructions

  • Come here to the front of the room (one at a time!)
  • You will look at my screen and answer a question
  • Please don’t say the answer out loud and don’t give hints to others!
  • Let’s see what happens! 😊


https://forms.gle/xUL39k7ngY2kXYJC7

Let’s discuss the results! 📊

What makes a research question good? 🤔

The importance of a good research question

  • The answer to a good research question should produce knowledge that people will care about
  • Addressing the question should (help) solve a problem, make a decision, or clarify/challenge our understanding of the world
  • But an interesting question is not enough
  • A good research design is a practical plan for research that makes the best use of available resources and produces a credible answer
  • The quality of a research design can be assessed by how well it produces results that can be used to guide policy and improve science:
    • A great research design produces results that clearly point in certain directions that we care about
    • A poor research design produces results that leave us in the dark — results with confusing interpretation

Source: University of Cambridge

Causality and experiments

  • As researchers, we are interested in research questions about how the world works

  • There are a number of different types of questions that we may want to answer. In academia, they are often divided into two broad categories:

    • Descriptive questions: Descriptions of a given phenomena: e.g., “how do teachers allocate their time during a school day?”
    • Causal questions: Questions about how \(X\) affects \(Y\): e.g., “Does providing vocational training to migrants improve their economic integration in the receiving country?”

  • Then we can move on to questions about why? \(\rightarrow\) i.e., knowing the effect of a cause is necessary before moving on to understanding the causes of an effect.

  • (Next sessions: more on about what we mean by causality and how experiments give us leverage to make causal claims.)

Theory

  • What is the phenomenon we want to explain?

    • Our outcome (we are going to call it \(Y\))

  • Does the cause we theorise lead to observing changes in \(Y\)?

    • Our treatment (in the context of experiments) (we are going to call it \(T\))
    • We will use \(X\) to refer to background variables (covariates)

  • What is the theory of change?

  • We are ultimately interested in how two theoretical concepts are related, measured by observed variables \(T\) (our treatment) and \(Y\) (our outcomes)

  • Why is theory important, then?

The importance of theory

  • There is no such thing as “just doing an experiment” 🧐

  • All research design involves theory, whether implicit or explicit

  • Our questions are value laden: For example, social scientists studied marijuana use in the 1950s as a form of “deviance”, the questions focused on “why are people making such bad decisions?” or “how can policy makers prevent marijuana use?”

  • Why do the research? We might want to change how scientists explain the world and/or change the policy decisions in (a) one place and time and/or (b) in other places and times

  • Research focused on learning the causal effect of \(T\) on \(Y\) requires a model of the world: how might intervention \(T\) might have an effect on some outcome \(Y\), and why, and how large might be the effect. It helps us think about how a different intervention or targeting different recipients might lead to different results.

  • Our theories and models are important not just for generating hypotheses, but for informing design and strategies for inference

  • Designing research will often clarify where we are less certain about our theories. Our theories will point to problems with our design. And questions arising from the process of design may indicate a need for more work on explanation and mechanism

Richard Feynman on the experimental method

Click on the image to watch the video! 📺

Designing or selecting your treatment

  • Operationalisation: The process of translating theoretical concepts into measurable variables.
    • Example: Turning the concept of “social isolation” into a measurable variable such as the frequency of social interactions
  • Key Questions:
    • How will we measure our outcomes?
    • What indicators will be used to represent the underlying concept?
    • How will we manipulate the cause of interest?
    • What intervention or treatment will be implemented?
  • Importance of Alignment:
    • The research design must align with the theoretical framework to ensure that the study is addressing the intended questions

Let’s consider the example from our experiment practicum

  • What is the outcome of interest (\(Y\))?

  • What is the cause of interest (\(T\))?

  • What can be a theory that yields to this experimental design?

  • What can be the main hypothesis?

  • How can we measure our outcomes?


  • What do you think?

Measuring treatments

  • Can we directly manipulate \(T\)? (underlying treatment concept of interest)

    • Ethical, logistical and other types of considerations can limit our ability to manipulate all of the indicators of \(T\)
    • At best, we may be able to change some of its indicators
    • We design a treatment, \(T\), to do so

  • How does T relate to \(T\)?

    • But T can be manipulating other things (bundled treatments)

  • Did everyone receive \(T\)?

    • Measure compliance

Source: World Health Organization, 2003

Thinking about the treatment from the practicum…

How do you feel about the world?

  • Now think of yourselves as the researchers

  • In pairs or groups of three:

    • Generate hypothesis on potential heterogeneous effects
    • Generate expected effect size
    • Discuss theories behind the hypothesis and expected effect size, with emphasis on the importance of theory
    • Other ways of measuring the outcome or mode of administering the treatment?

Measuring outcomes

  • We often cannot directly observe the true value of the outcome concept for most of the outcomes we are interested in

  • Examples:

    • Correct answers to problems (indicators) for underlying mathematical aptitude (the actual phenomenon)
    • Days without food (indicators) for hunger (the actual phenomenon)
    • Reports of bribes (indicators) for corruption (the actual phenomenon)

  • Moreover, the underlying outcome concept may be even under debate (e.g., democracy)

  • If our indicators don’t measure the underlying concept that we’re interested in, then we may not be able to learn very much, even if we have an otherwise very sound experiment

To sum up:

  • Articulate and refine your research question:
    • Clearly define the problem you are addressing
    • Consider the implications of different possible answers
  • Develop your research design:
    • Plan your intervention and data collection methods
    • Consider the resources available and the feasibility of the design
  • Plan your analysis:
    • State and justify your hypotheses
    • Pre-register your analysis plan to enhance credibility
  • Implement your intervention and collect data:
    • Execute your research design
    • Ensure data quality and compliance with the intervention
  • Analyse your data and write up your results:
    • Conduct statistical analyses
    • Interpret the findings in light of the research question and theoretical framework

Research Design